Infrastructure and architecture for development and execution of predictive models

ABSTRACT

A system that enables development and execution of predictive models comprises a centralized data management system, a data extraction tool a model validation tool and a model execution tool. In embodiments, a data management system includes a data management server that can be accessed via a web browser that stores data. An extraction tool includes a data filter adapted to filter data based on, for example, a population criteria, a sample size, and a date range criteria. A model validation tool validates the model. A model execution tool allows a user to score the model.

RELATED APPLICATIONS

This patent application is a continuation of and claims priority to, andthe benefit of, U.S. patent application Ser. No. 12/100,225, entitled“INFRASTRUCTURE AND ARCHITECTURE FOR DEVELOPMENT AND EXECUTION OFPREDICTIVE MODELS” filed on Apr. 9, 2008, now U.S. Pat. No. 7,953,762B2, which is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to predictive modeling, and specificallyto reducing the time and resources required for developing, deploying,and executing a model.

2. Background of Invention

Direct marketing campaigns involve targeting a set of existing customersfor a promotional offer. For example, credit card customers that chargea certain amount on travel purchases per month can be offered a creditcard with travel rewards, because it is likely that they will accept theoffer. Decision sciences and predictive modeling are used to estimatethe likelihood (referred to herein as a score) that a particularcustomer will accept an offer. Thus, the effectiveness of a directmarketing campaign is related to the robustness of the predictive modelon which it was based.

Because models are so integral to direct marketing campaigns, it isdesirable to execute multiple models efficiently while reducing the timeand effort it takes for model development and model deployment. Inexisting systems, there are deficiencies in model development, modeldeployment, and model execution. These deficiencies increase the timeand cost of developing a model and how effectively a model can beexecuted.

In existing systems, data is stored in multiple disparate sources ofcustomer data each having their own unique definitions and accessrequirements. When data is stored in multiple disparate sources, amodeler needs to standardize data and create data tables before data canbe analyzed. This results in inconsistent results, compliance risks, andan overall low confidence in the outcomes.

In existing systems, implementing the model logic in the developmentphase is not a seamless process. In existing systems, models are oftendeveloped in one programming language and deployed in anotherprogramming language. Thus, implementing a model requires convertinglogic code into system compatible code. Converting model logic into amodel that can be implemented requires using resources with hightechnical skills to translate logic into system compatible code. Thiscan result in numerous errors and prolonged implementations whichincreases the time it takes for the deployment phase to be completed.

In some existing systems, model execution occurs at a mainframe locationand is based on billing cycles. That is, customers are scored at the endof their billing cycle. Because customer billing cycles vary across amonth (e.g. customer A's billing cycle ends the 15^(th) and customer B'sbilling cycle ends the 30^(th)), it takes a full month to score acustomer base. This is problematic because customer scores may changedaily. Further, scores for the customers at the beginning of the billingcycle may be obsolete by the time the entire customer base is scored.

What is needed is an end-to-end integrated process of model development,model deployment, and model execution for customer marketing campaignsthat enables rapid development and dynamic execution of models.

SUMMARY OF THE INVENTION

An infrastructure and architecture according to an embodiment of thepresent invention enables the rapid development and execution ofpredictive models. Exemplary embodiments provide solutions in the systemdesign and process flow integration of model development,implementation, and execution for customer marketing campaigns.

Exemplary embodiments enable the rapid implementation of customermarketing models, rules, and segmentation strategies by logicallyintegrating the three phases of customer marketing modeling. Thisenables significant reduction in the time-to-market of new strategies byreducing model deployment time. This reduced time-to-market allowscompanies to learn and respond to marketplace changes faster (e.g.adjust balance-transfer pricing to reflect reduction in prime rate).

In the execution phase, rapid assessments can occur using current datareadily usable on demand. Further, exemplary embodiments can executemultiple models in a short time period using the most current customerdata. This ensures ever-ready “scores” for customer marketing solutions.Exemplary embodiments can include quality controls built into theexecution process that proactively track and notify input data anomaliesand model volatility. Increases in model coverage across variouschannels/products complemented by rich, fresh, and accurate data providemajor improvements in response rates and ultimately generate higherreturns. For example, models may incorporate customer contact andproduct preferences enabling customized offers that generate higherresponse. Execution in exemplary embodiments allows businesses torespond immediately to changes in customer behavior. For example,customers who start showing an improved credit profile can be targetedselectively, in a much faster manner, for line increase offers.

In an embodiment, a system that enables development and execution ofpredictive models includes a centralized data management system, a dataextraction tool, a model validation tool and a model execution tool.

In an embodiment, a data management system includes a data managementserver that can be accessed via a web browser that stores data in theform of a flat file. Further, a flat file may be a SAS data set.

In an embodiment, the data management system includes a data warehousethat stores data in the form of SQL variables.

In an embodiment, data to be extracted by the extraction tool appears inthe form of a SAS data set and the extraction tool converts a SQLvariable to a SAS variable.

In an embodiment, the extraction tool includes a privacy protectionfunction that restricts user access to extracted data.

In an embodiment, the extraction tool includes a data filter adapted tofilter data based on a population criteria, a sample size, and a daterange criteria. Further, the data filter may filter data by combiningcriteria using logical operands. The data filter may also include asearchable data dictionary. The data filter can filter data based on thecontents of an uploaded file.

In an embodiment, the model validation tool includes an intelligent codeparser that checks a model for errors. Further, the model validationtool can validate a model by scoring an analytical environment data setand a production environment data set, and compare scores using areport. The model validation tool can validate a model using apre-defined test case.

In an embodiment, the model execution tool allows a user to select whenand how often a model is scored. The model execution tool can include anexpiry date function that allows a user to specify how long a model willrun. The model execution tool can score models using a SAS score engine.

In an embodiment, the model execution tool creates a plurality ofvariable files, each variable file referencing data variables used inmodels, wherein the variables referenced in a variable file aredetermined by a number of models that use a particular variable.

In an embodiment, the execution tool tracks model behavior against abaseline measure and alerts a user if model behavior deviates from thebase line.

Further embodiments, features, and advantages of the invention, as wellas the structure and operation of the various embodiments of theinvention are described in detail below with reference to accompanyingdrawings.

BRIEF DESCRIPTION OF THE FIGURES

The present invention is described with reference to the accompanyingdrawings. In the drawings, like reference numbers indicate identical orfunctionally similar elements.

FIG. 1 provides a block diagram of an exemplary embodiment of a systemfor model development, deployment, and execution.

FIG. 2 provides an exemplary embodiment of a system for modeldevelopment, deployment, and execution.

FIG. 3 provides a block diagram of an exemplary model extraction tool.

FIG. 4 provides a block diagram of an exemplary model validation tool.

FIG. 5 provides a block diagram of an exemplary model execution tool.

FIG. 6 is a diagram of a computer system on which the methods andsystems herein described can be implemented.

FIG. 7 is a flowchart depicting an exemplary embodiment of a process formodel development, deployment, and execution.

DETAILED DESCRIPTION OF THE INVENTION

While the present invention is described herein with reference toillustrative embodiments for particular applications, it should beunderstood that the invention is not limited thereto. Those skilled inthe art with access to the teachings provided herein will recognizeadditional modifications, applications, and embodiments within the scopethereof and additional fields in which the invention would be ofsignificant utility.

Model development occurs in what is called the analytic environment andinvolves building a model from customer data. Model developmentinvolves, without limitation, gathering customer data, analyzingcustomer data, creating model logic based on the analysis, andimplementing the model logic. Gathering customer data typically involvesgenerating and maintaining variables (e.g., household size, wallet size,income, travel expenses, etc.) using customer records. Systems mayemploy thousands of customer variables for tens of millions ofcustomers. Analyzing customer data may involve analyzing customervariables to determine trends in customer behavior (e.g. customers witha higher income spend a greater percentage of their income on travel).Creating model logic may involve determining which of the thousands ofvariables will be used in a model and how they will be used to determinethe likelihood a customer will behave in a certain way. For example, ina simple model, the likelihood of acceptance could be determined by theweighted average of a dozen variables. Implementing the model logic mayinvolve putting the model into a form such that the model can beimplemented in the development phase.

Model deployment may occur in what is called the production environmentand involves testing the accuracy of the model against customer data todetermine if the model contains any bugs and if the model achieves theexpected results. Model deployment may involve actual customer scoringusing previously-received marketing results.

Model execution involves using the model on live customer variablevalues where the results of the model are used to determine if an offershould be sent to a customer.

FIG. 1 provides a block diagram of an exemplary embodiment of a system102 for model development, deployment, and execution. System 102includes network 100, data sources 200, data management system 300, userstation 400, extraction tool 500, validation tool 600, model library700, execution tool 800, and population stability index (PSI) tool 900.It should be noted that although functional elements are illustrated asdistinct blocks in FIG. 1, functional elements should not be construedas being distinct physical structures. Reference is made to FIG. 2 whichillustrates system 102 embodied in a computer network architecture. Itshould be noted that although system 102 is described with respect tothe credit card industry, the predictive modeling system can be used inmany different industries, including, for example and withoutlimitation, weather, retail, web search, transportation, communications,healthcare, hiring, and finance. Additionally, although extraction tool500, validation tool 600, model library 700, execution tool 800, and PSItool 900 are illustrated as embodied as being located at distinctlocations on a computer network, one of ordinary skill in the art willrecognize that this is for descriptive purposes only, as extraction tool500, validation tool 600, model library 700, execution tool 800, and PSItool 900 can be co-located anywhere on a computer network. For example,extraction tool 500, validation tool 600, and execution tool 800 may belocated on a user station, such as user station 400 and model library700 may be located on a central server, without departing from thespirit and scope of the present invention.

Network 100 is a communication network that allows communication tooccur between the elements in the system. In the exemplary embodiment ofFIG. 2, network 100 is illustrated as comprising a data network 100 aand user network 100 b. Although network 100 is shown as two networks inFIG. 2 network 100, network 100 can be any network or combination ofnetworks that can carry data communication, and may be referred toherein as a computer network. Such network 100 can include, but is notlimited to, a local area network, medium area network, and/or wide areanetwork such as the Internet. Network 100 can support protocols andtechnology including, but not limited to, World Wide Web protocolsand/or services. Intermediate web servers, gateways, or other serversmay be provided between components of system 102 depending upon aparticular application or environment.

Data sources 200 include systems that are sources of customer data. Forexample, data sources 200 may include data from the following exemplarysystems: authorization systems, risk management systems, demographicsystems, spend/bill systems, etc. In the exemplary embodiment of FIG. 2,data sources 200 are illustrated as multiple customer data servers 200 aand 200 b. Each data source 200 may have its own unique definitions andaccess requirements.

Data management system 300 processes and standardizes the data receivedfrom the various data sources 200. Processing and standardizing data mayinvolve validating received data to insure the data does not containerrors, aggregating the received data to create aggregated data, andprofiling the aggregated data so data trends across the customer basecan be recognized. In one exemplary embodiment, data management system300 uses software developed by the Ab Initio Software Corporation ofLexington, Mass. to perform the data processing and standardization.Data management system 300 provides centralized data that can bedistributed across the system.

In the exemplary embodiment of FIG. 2, data management system 300includes data management server 302 and optional data warehouse 304.Data management server 302 receives data from customer data servers 200a and 200 b via data network 100 a. Data management server 302 creates,for example, one feed per data source in the form of a flat file.

In an embodiment where a data warehouse 304 is used, data managementserver 302 may send the flat file to data warehouse 304. Data warehouse304 converts the flat file into, for example, relational database tableelements for storage. Data warehouse 304 can store thousands of dataparameters for tens of millions of customers. In an exemplaryembodiment, data is stored in data warehouse 304 in the form of SQL(Structured Query Language) variables. Data management system 300 canalso include metadata dictionaries.

User station 400 provides an interface through which a modeler candevelop, deploy, and execute a model. In the exemplary embodiment ofFIG. 2, user station 400 is a computer with a web browser that allows amodeler to develop, deploy, and execute a model via a web portal.

Extraction tool 500 is used by a modeler to complete the developmentphase of a model. In the exemplary embodiment of FIG. 2, extraction tool500 is implemented in the form of an extraction server 500 that resideson developer network 100 b. FIG. 3 provides a block diagram of anexemplary extraction tool 500. As shown in FIG. 3, model extraction tool500 includes user interface 502, SQL Macro 504, privacy function 506,filtering function 508, and model building function 510.

Extraction tool 500 provides a modeler with a user interface 502 thatallows a modeler to develop a model. In the exemplary embodiment, userinterface 502 is a standard GUI (Graphical User Interface). Modeldevelopment involves gathering customer data, analyzing customer data,creating model logic based on the analysis, and implementing the modellogic.

As described above, data gathering may be completed by data managementsystem 300. Thus, in an exemplary system, data gathering is transparentto the modeler. As such, the modeler need not standardize data norcreate data tables before data can be analyzed. In an exemplaryembodiment, variables on extraction tool 500 appear in the form of SASdatasets. The SAS variable names are standardized and consistent.

It should be noted that in some embodiments where data in the datamanagement system 300 exists in the form of SQL variables on datawarehouse 304, extraction tool includes a processing SQL macro 504 thatconverts SQL variables to SAS variables. This allows the end-user towork with SAS variables and avoid performing complex queries and joins.Thus, the modeler works with SAS datasets that can be directly pluggedinto a model.

In an embodiment where data is stored on data management server 302 asflat files, the flat files can be SAS flat files. Thus, SAS datasetextracts can be delivered directly to modelers on demand. The flat filearchitecture enables the optimization of system resources and providesimprovements in the performance, efficiency, scalability, anduser-experience of the extraction process compared to existing systems.

Further, the flat file architecture enables faster implementation of newdata by removing the need to transform native data elements torelational database elements. In addition, there is no need to predefineall fields that will be stored. If the full source dataset is stored,new fields can be easily added by adding them to the extraction tool500.

Flat file data storage typically costs less than data storage in awarehouse due to compression and lower indexing overhead. In addition,data warehouse 304 can be eliminated and all extracts run on datamanagement server 302.

The ability to retain data in its native format without transforming itinto relational database tables also enhances data compliance andsecurity. Data does not have to be transferred from the data managementserver 302 to the data warehouse 304. Also, restrictions on accessingdata directly on the data management server 302 may be in place, toincrease data security for sensitive and confidential customerinformation. Management Information System (MIS) tools can also beembedded into data management server 302 to track capacity usage by datasource, market, variable, organization, and/or modeler to allowmanagement of the platform for maximum return. Further, an issuetracking database (not shown) may provide transparency reporting metricsto manage and optimize the performance and efficiency of extraction tool500.

It should be noted that when a flat file architecture is used, thesystem does not significantly degrade as the amount of data it holdsincreases. In contrast, relational database warehouse performance maydeteriorate as new amounts of data increases in the aggregate.

Privacy protection function 506 may be included to protect the privacyof the consumers. In an embodiment, privacy protection function 506gives the modeler a temporary password when a modeler requests access toa data set. The temporary password allows access to the specific datasetand the ability to copy it to a user directory. Further, only authorizedmodelers may be able to access extraction tool 500.

Extraction tool 500 may be equipped with parallel processing andintelligent partitioning, prioritization, and routing capabilities. Thisis an alternative to data extraction processes that follow a sequentialloading paradigm, which causes congestion and delays in the dataextraction process. With parallel processing, multiple jobs can run atthe same time and not create a backlog of requests. In addition,intelligent partitioning, prioritization, and routing capabilitiesallocate jobs based on estimated size, targeted population, andvariables selected so multiple jobs can run against the databasesimultaneously and optimize CPU throughput.

Filtering function 508 allows a modeler to analyze a subset of the datastored by data management system 300. As noted above, data managementsystem 300 may store thousands of variables for tens of millions ofcustomers. Filtering function 508 may allow a modeler to filter databased on a population criteria. For example, a user can filter based ona credit card type, credit limit, household size, etc. Filteringfunction 508 may allow a modeler to filter data based on a sample sizecriteria. For example, a modeler can select a sample size of 0.1%, 1%,5%, 10%, 20%, or 100% of the customer base. Further, a modeler may beable to upload a customer file of customers, allowing the filteringfunction 508 to return data for the customers in the file.

Filtering function 508 may have the ability to perform mathematical andlogical operands on variables to segment a population in the dataextract. For example, a modeler may specify all users with a creditlimit of less than $10,000 and with a household size of less than two.Filtering function 508 may provide the modeler the ability to selecthistory and date ranges. For example, a modeler may specify all usersthat charged over $1000 from September 2003-October 2003.

Filtering function 508 may also include a searchable online datadictionary with policy usage criteria for each variable. Further,filtering function 508 may provide advanced searching features forvariables. A modeler may also upload variables from a file and savevariables to a file.

Once a modeler analyzes data, the modeler may create model logic basedon the analysis using model building function 510. A modeler can createmodel logic by selecting the variables to be included in the model andspecify how the variables will be used in the model to score a customer.Model logic may be created by a modeler writing model code for a model.In an exemplary embodiment, the model code is SAS code.

Model building function 510 allows a modeler to run a simulation of amodel on customer data before the model is complete. Further, modelbuilding function 510 may allow a modeler to see a sample output of amodel before the model is complete. Model building function 510 mayallow a modeler to terminate a simulation or re-run a simulation whileit is in progress. Further, simulations may be stored and shared. Modelbuilding function 510 may provide a modeler the ability to, for exampleand without limitation, save jobs as drafts, create a summary and aconfirmation before submittal, extract jobs across multiple months andreturn in one dataset, estimate the size of output at job submissiontime, estimate the time of when a job will be completed, track thepercent of completion, and/or upload and filter scrambled information.Further, in model building function 510, scored and rescored datasetsmay be combined into one consolidated dataset. Once a modeler issatisfied with a model, a modeler may submit the model. At this point,the model development phase is complete and the model deployment begins.

Validation tool 600 may be used by a modeler to complete the developmentphase of a model. In the exemplary embodiment of FIG. 2, validation tool600 is implemented in the form of a validation server that resides ondeveloper network 100 b. FIG. 4 provides an exemplary block diagram ofvalidation tool 600. As shown in FIG. 4, validation tool 600 includesuser interface 602, intelligent code parser 604, and model validationprocess 606.

User interface 602 provides a modeler with an interface that allows amodeler to complete deployment of a model. In an exemplary embodimentinterface 602 is a standard GUI interface.

Intelligent code parser 604 checks for and rejects erroneous code priorto deployment.

Model validation process 606 may provide all independent variables and“scores” for sign-off. Model validation process 606 allows a modeler torun a validation dataset against up to 100% of the data prior todeployment. This allows the modeler to validate the model and scoresproduced by the model prior to execution in the production environment.Further, model validation process 606 may allow the modeler to specifyparameters on which customers should be scored, where the scores shouldbe sent, and how frequently the scores should be updated. Further, amodeler can change priorities, and/or set limits on others wishing touse a model. Once these parameters are entered, then the model may runautomatically without the need of manual intervention.

In an exemplary embodiment, model validation process 606 uses Syncsortsoftware available from Syncsort Incorporated, of Woodcliff Lake, N.J.,as an ETL (Extract Transform Load) tool that provides joining andloading of data from multiple files into one dataset for model logicdeployment.

In an embodiment, model validation provides model scores in mainframeand warehouse variables in one dataset.

In an embodiment, validation reports are automated. That is, modelvalidation process 606 captures the logs and profile reports of thevalidation datasets in both the production environment and theanalytical environment. The two validation datasets are then measured tovalidate and ensure consistency across environments. Automating thiseliminates the need for the modeler to manually validate a model.

In an embodiment, model validation process 606 includes advancedtransparency tools to dissect model code logic in order to ensure bestpractices and optimize efficiencies. That is, model validation process606 may give the modeler the capability to automatically obtain the codelog and process steps. This allows economies of both time and scale tobe achieved through standardization and sharing/leveraging bestpractices.

Validation process 606 may allow a modeler to create a list ofpre-defined User-Acceptance Testing (UAT) cases that may be used byother modelers, thereby accelerating and automating the testing process.

Model validation process 606 may include a standardized list ofvalidation rules to ensure comprehensive data testing procedures priorto implementation. Model validation process 606 may also include arequest initiation, requirement gathering and data management checklistprocesses. Further, model validation process 606 may also includepre-approved policy approval for bulk data sources to eliminate manualsteps and remove policy approval from a critical path. Model validationprocess 606 may allow creation of an audit trail and approval forrequests. Once a modeler is satisfied with the results from validationtool 600, the modeler may deploy the model.

Model library 700 stores models that have been deployed by variousmodelers. In the exemplary embodiment of FIG. 2, model library 700 isimplemented in the form of a model library server that resides ondeveloper network 100 b. At any given time, model library 700 caninclude thousands of models. In an exemplary embodiment, models arestored in the form of SAS models. Model library 700 may provide modelersthe ability to update a model using a web interface after the model hasbeen deployed. A robust quality and control process can be maintained toensure compliance standards are strictly enforced in model deployment.The quality and control process can provide different access levels andrights depending on modeler credentials. The quality and control processcan also provide an approval mechanism that requires a model to beapproved before the model can be deployed.

Execution tool 800 scores deployed models based on the customer datastored in data management system 300. FIG. 5 provides a block diagram ofan exemplary execution tool 800. As shown in FIG. 5, execution tool 800includes score engine 802 and data pivoting function 804.

Score engine 802 is used to score models. In an exemplary embodiment,SAS is used as an engine to score models. When a model is being scored,execution tool 800 allows modelers to select when and how often theirmodels will be scored. In an embodiment, a modeler can select when andhow often their models are scored at any time during execution. Themodel can then run on an automated “lights-out” schedule. In general,“lights-out” refers to operation that is constantly occurring.

Problems can arise when obsolete models continue to run and consumemassive amounts of system resources. Thus, in an embodiment, rather thanmodels running lights-out on pre-defined automated intervals, a modelermay call models as needed to run for use in campaigns. Further, anexpiry date function can be implemented. An expiry date function allowsmodelers to specify how long they want their model to run in theproduction environment.

In an embodiment where score engine 802 uses SAS, SAS parallelizationallows multiple jobs to run in parallel as opposed to sequentially. Thisincreases the performance and speed of model execution. Using the systemof the exemplary embodiment, model scoring may occur using current datareadily available on demand. Further, execution tool 800 may useresource utilization via parallel scheduling, intelligent data pivotingand optimization.

Data pivoting function 802 is targeted towards optimizing how data feedsare accepted from data management system 300. In an embodiment, duringexecution, execution tool 800 accepts the entire input file of all thevariables for all of the customers in the data management system 300.However, most models use only a small portion of all of the variables.Further, the combination of the models being executed use only a smallportion of all the variables. Thus, in an embodiment, data pivotingfunction 802 is used to improve the cycle time to score a largepopulation by only utilizing active variables.

Data pivoting function 802 is structured on a self-learning process.When the data feeds are received from the data management system 300(e.g. daily/weekly), data pivoting function 802 looks at all the “lightsout” models in production and determines how many models need aparticular variable from a given feed. Based on this information, theprocess breaks the input feed into multiple data files. Thedetermination of how many split files need to be created and how manyvariables need to be placed in each split file is dynamically controlledthrough a configuration file.

An example configuration file may specify storing variables from aparticular feed into split files as follows: storing all unusedvariables in a first split file, storing all variables used less than 9times in a second split file, storing all variables used between 10 and50 times in a third split file, and storing all variables used more than50 times from a feed in a fourth split file.

If a particular file is not yielding expected results, then theparameters in the configuration file can be changed to change the splitprocess for the next time data is received from data management system300. Because the manner in which a feed is split for optimization isdependent upon the usage by models, data reflecting the number of splitfiles and the variables residing on each split file at any given timemay be maintained.

In the exemplary embodiment, model scores reside in a flat filearchitecture. Further, data related to model performance may bemaintained for speed, accuracy, flexibility, and optimal maintenance.Data pivoting function 802 may allow only active variables in themetadata to be passed to the production environment to allow greaterspeed and execution.

Population Stability Index (PSI) tool 900 allows users to analyze andtrack the behavior and performance of models over time. In the exemplaryembodiment of FIG. 2, PSI 900 is implemented in the form of softwarethat resides on execution server 800. A baseline measure of the model isestablished and each consecutive run of that model is measured againstthat baseline. PSI tool 900 tracks the deviation of that model overtime. If there is a statistically significant change, PSI tool 900 willcapture that change and automatically alert the modeler.

Computer System Implementation

The present invention (i.e., system 102 or any part(s) or function(s)thereof) may be implemented using hardware, software or a combinationthereof and may be implemented in one or more computer systems or otherprocessing systems. However, the manipulations performed by the presentinvention were often referred to in terms, such as adding or comparing,which are commonly associated with mental operations performed by ahuman operator. No such capability of a human operator is necessary, ordesirable in most cases, in any of the operations described herein whichform part of the present invention. Rather, the operations are machineoperations. Useful machines for performing the operation of the presentinvention include general purpose digital computers or similar devices.

In fact, in one embodiment, the invention is directed toward one or morecomputer systems capable of carrying out the functionality describedherein. An example of a computer system 1000 is shown in FIG. 6.

The computer system 1000 includes one or more processors, such asprocessor 1004. The processor 1004 is connected to a communicationinfrastructure 1006 (e.g., a communications bus, cross over bar, ornetwork). Various software embodiments are described in terms of thisexemplary computer system. After reading this description, it willbecome apparent to a person skilled in the relevant art(s) how toimplement the invention using other computer systems and/orarchitectures.

Computer system 1000 can include a display interface 1002 that forwardsgraphics, text, and other data from the communication infrastructure1006 (or from a frame buffer not shown) for display on the display unit1030.

Computer system 1000 also includes a main memory 1008, a tangible,non-transitory memory, preferably random access memory (RAM), and mayalso include a secondary memory 1010. The secondary memory 1010 mayinclude, for example, a hard disk drive 1012 and/or a removable storagedrive 1014, representing a floppy disk drive, a magnetic tape drive, anoptical disk drive, etc. The removable storage drive 1014 reads fromand/or writes to a removable storage unit 1018 in a well known manner.Removable storage unit 1018 represents a floppy disk, magnetic tape,optical disk, etc. which is read by and written to by removable storagedrive 1014. As will be appreciated, the removable storage unit 1018includes a non-transitory, tangible computer storage medium havingstored therein computer software and/or data.

In alternative embodiments, secondary memory 1010 may include othersimilar devices for allowing computer programs or other instructions tobe loaded into computer system 1000. Such devices may include, forexample, a removable storage unit 1022 and an interface 1020. Examplesof such may include a program cartridge and cartridge interface (such asthat found in video game devices), a removable memory chip (such as anerasable programmable read only memory (EPROM), or programmable readonly memory (PROM)) and associated socket, and other removable storageunits 1022 and interfaces 1020, which allow software and data to betransferred from the removable storage unit 1022 to computer system1000.

Computer system 1000 may also include a communications interface 1024.Communications interface 1024 allows software and data to be transferredbetween computer system 1000 and external devices. Examples ofcommunications interface 1024 may include a modem, a network interface(such as an Ethernet card), a communications port, a Personal ComputerMemory Card International Association (PCMCIA) slot and card, etc.Software and data transferred via communications interface 1024 are inthe form of signals 1028 which may be electronic, electromagnetic,optical or other signals capable of being received by communicationsinterface 1024. These signals 1028 are provided to communicationsinterface N24 via a communications path (e.g., channel) 1026. Thischannel 1026 carries signals 1028 and may be implemented using wire orcable, fiber optics, a telephone line, a cellular link, an radiofrequency (RF) link and other communications channels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to media such as removablestorage drive 1014, a hard disk installed in hard disk drive 1012, andsignals 1028. These computer program products provide software tocomputer system 1000. The invention is directed to such computer programproducts.

Computer programs (also referred to as computer control logic) arestored in main memory 1008 and/or secondary memory 1010. Computerprograms may also be received via communications interface 1024. Suchcomputer programs, when executed, enable the computer system 1000 toperform the features of the present invention, as discussed herein. Inparticular, the computer programs, when executed, enable the processor1004 to perform the features of the present invention. Accordingly, suchcomputer programs represent controllers of the computer system 1000.

In an embodiment where the invention is implemented using software, thesoftware may be stored in a computer program product or an article ofmanufacture and loaded into computer system 1000 using removable storagedrive 1014, hard drive 1012 or communications interface 1024. Thecontrol logic (software), when executed by the processor 1004, causesthe processor 1004 to perform the functions of the invention asdescribed herein.

In another embodiment, the invention is implemented primarily inhardware using, for example, hardware components such as applicationspecific integrated circuits (ASICs). Implementation of the hardwarestate machine so as to perform the functions described herein will beapparent to persons skilled in the relevant art(s).

In yet another embodiment, the invention is implemented using acombination of both hardware and software.

CONCLUSION

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample, and not limitation. It will be apparent to persons skilled inthe relevant art(s) that various changes in form and detail can be madetherein without departing from the spirit and scope of the presentinvention. Thus, the present invention should not be limited by any ofthe above described exemplary embodiments, but should be defined only inaccordance with the following claims and their equivalents.

In addition, it should be understood that the figures and screen shotsillustrated in the attachments, which highlight the functionality andadvantages of the present invention, are presented for example purposesonly. The architecture of the present invention is sufficiently flexibleand configurable, such that it may be utilized (and navigated) in waysother than that shown in the accompanying figures.

Further, the purpose of the foregoing Abstract is to enable the U.S.Patent and Trademark Office and the public generally, and especially thescientists, engineers and practitioners in the art who are not familiarwith patent or legal terms or phraseology, to determine quickly from acursory inspection the nature and essence of the technical disclosure ofthe application. The Abstract is not intended to be limiting as to thescope of the present invention in any way.

1. A computer-implemented method comprising: simultaneously generatingand executing, by a computer-based system for generating a plurality ofpredictive models, the plurality of predictive models, wherein at leastone of the predictive models in the plurality of predictive models isbased upon analysis and manipulation of data associated with a group ofconsumers, and wherein each model is given a score representing anaccuracy of the model; and tracking, by the computer-based system, abehavior and performance of the plurality of predictive models using aPopulation Stability Index (“PSI”).
 2. The method of claim 1, furthercomprising transmitting, by the computer-based system, to each consumerin a group of consumers a score that indicates a likelihood that eachconsumer in the group will respond to an offer.
 3. The method of claim1, further comprising displaying, by the computer-based system and on auser interface, the data associated with the group of consumers.
 4. Themethod of claim 1 further comprising manipulating, by the computer-basedsystem, the data by at least one of: filtering the data based on asample size, filtering the data based on an uploaded file, filtering thedata using mathematical and logical operations, and filtering the databased on a credit limit.
 5. The method of claim 1, further comprisingscoring during specified intervals the predictive model, wherein thescore represents the accuracy of the model.
 6. The method of claim 1,wherein the data is queried using Statistical Data Analysis (“SAS”)code.
 7. The method of claim 1, wherein the data is queried usingStructured Query Language (“SQL”) queries.
 8. The method of claim 1,further comprising displaying, by the computer-based system and using asearchable data dictionary, policy usage criteria associated with thedata.
 9. The method of claim 1, further comprising checking, by thecomputer-based system and using an intelligent code parser, thepredictive model for errors.
 10. The method of claim 1, furthercomprising collecting and standardizing the data.
 11. The method ofclaim 1, further comprising optimizing, by the computer-based system,how the data are accepted by way of a self-learning process.
 12. Themethod of claim 1, further comprising validating, by the computer-basedsystem, the predictive model prior to deployment of the predictivemodel.
 13. The method of claim 1, further comprising presenting, by thecomputer-based system and through a user interface, a standardized listof rules associated with a data testing procedure.
 14. The method ofclaim 1, further comprising updating, by the computer-based system andthrough a user interface, the predictive model after the predictivemodel has been deployed.
 15. The method of claim 1, further comprisingmaintaining, by the computer-based system, a library that includes aplurality of the predictive models.
 16. The method of claim 1, furthercomprising grouping, by the computer based system, a plurality ofconsumers based on a spending attribute associated with the plurality ofconsumers to create the group of consumers.
 17. The method of claim 1,further comprising receiving, by the computer-based system and throughthe user interface, second data associated with a second group ofconsumers and at least one of: a score destination and an updateinterval.
 18. An article of manufacture including a non-transitory,tangible computer readable storage medium having instructions storedthereon for generating a plurality of predictive models that, inresponse to execution by a computer-based system for generating theplurality of predictive models, cause the computer-based system toperform operations comprising: simultaneously generating and executing,by the computer-based system, the plurality of predictive models,wherein at least one of the predictive models in the plurality ofpredictive models is based upon analysis and manipulation of dataassociated with a group of consumers, and wherein each model is given ascore representing an accuracy of the model; and tracking, by thecomputer-based system, a behavior and performance of the plurality ofpredictive models using a Population Stability index (“PSI”).
 19. Acomputer system comprising: a tangible, non-transitory memorycommunicating with a processor for generating a plurality of predictivemodels, the tangible, non-transitory memory having instructions storedthereon that, in response to execution by the processor, cause theprocessor to perform operations comprising: simultaneously generatingand executing, by the processor, the plurality of predictive models,wherein at least one of the predictive models in the plurality ofpredictive models is based upon analysis and manipulation of dataassociated with a group of consumers, and wherein each model is given ascore representing an accuracy of the model; and tracking, by theprocessor, a behavior and performance of the plurality of predictivemodels using a Population Stability Index (“PSI”).